Telemetry by anthonyshull · Pull Request #21 · mbta/technology-docs

anthonyshull · 2024-02-07T20:42:31Z

No description provided.

paulswartz

Thanks for writing this up! This is a great overview of the approach.

paulswartz · 2024-02-15T15:34:46Z

+  3. **Traces** - are the representations of individual requests or transactions that flow through a system, which can help identify bottlenecks, dependencies, and root causes of issues.
+
+TID engineers currently make extensive use of logging.
+But, we don't currently have an established mechanism for collecting metrics or traces.


suggestion: we do have an "established mechanism for collecting metrics" at least, based on the definition above ("numerical measurements of system performance and behavior"). We have CPU/memory usage, response times, &c, already, via structured logs.

The fact that we can log numerical values does not make them metrics. Metrics are stored and accessed differently, so they must be placed in a metric-specific index.

Is there another definition of metrics that talks about their storage and access? The definition you used above doesn't appear to include that.

paulswartz · 2024-02-15T15:39:00Z

+- You are concerned about your ECS resource usage.
+
+The UF sidecar uses CPU and memory resources.
+Though the footprint is small, you might be worried that increasing your compute usage might affect the rest of your cluster.


question: do we know what the footprint is for UF running in ECS?

These are the published requirements: https://docs.splunk.com/Documentation/Forwarder/9.2.0/Forwarder/Deploy. For dotcom, we use 4 3GHz processors and 16GB of RAM. So, the UF would be 1.5/12 12.5% of the CPU and 3% of RAM.

It's a sidecar though, correct? So it's additional resources (not many it sounds like), but some.

paulswartz · 2024-02-15T15:42:53Z

+But, as noted above, logs are just one of three tools used to observe systems.
+Metrics are optimized for storage and retrieval as numbers in a time series, which makes them cheap to store and fast to retrieve.
+
+By adopting metrics, we can speed up slow queries, decrease the time necessary to generate dashboards and reports, and make previously impossible alerts possible.


question: do you have examples of "impossible alerts"?

Yeah, let's say you want to know if the number of requests you received over the last minute was anomalous. You would want to compare that minute to other bins over the past week. Looking at logs over the past week takes too long...because they aren't optimized to be viewed that way.

We have a alert which does this for another application already: search for "Concentrate: Unusual Metric Drop" in the Splunk alerts.

paulswartz · 2024-02-15T15:43:28Z

+
+By adopting metrics, we can speed up slow queries, decrease the time necessary to generate dashboards and reports, and make previously impossible alerts possible.
+
+Furthermore, we can decrease costs by reducing the data we send to Splunk.


question: does reducing the data we send to Splunk reduce our costs? It would also be interesting to compare with the additional resources needed to run the UF sidecar.

We pay for ingestion. Ingesting a log for every request (150 per second * 60 seconds) versus one metric for a minute is a 9,000:1 ratio.

Are we going to stop logging a message for each request?

paulswartz · 2024-02-15T18:44:28Z

+But, as noted above, logs are just one of three tools used to observe systems.
+Metrics are optimized for storage and retrieval as numbers in a time series, which makes them cheap to store and fast to retrieve.
+
+By adopting metrics, we can speed up slow queries, decrease the time necessary to generate dashboards and reports, and make previously impossible alerts possible.


question: do we have a comparison about how much time it takes currently to make dashboards, and how much it'll take with the new metrics collection approach?

This doesn't mean how long it takes to create the dashboard but to render it.

I haven't seen an issue with dashboard queries taking a long time, so I was hoping you had examples of where metrics would measurably improve things. For example, the API and Dotcom dashboards seems to load quickly enough?

paulswartz · 2024-02-15T19:25:55Z

+# Motivation
+
+Logs are handy for debugging and other unstructured research like a security audit.
+However, many TID developers have noted that some queries based on logs are incredibly slow, if not impossible, to run.


suggestion: it would be good to include some examples here.

If you take your example below and run it over a week. Took me 2 minutes and 30 seconds. Now, imagine loading a dashboard looking over the past week. It takes minutes to load. This is a real-world example from when I tried to find a good day/time to release the new caching work where we would have the fewest visitors and, therefore, incur the least risk.

It's 3:30 - 4am, and it took under 10 seconds for 7 days worth of data; 30 days took 25 seconds. I'm sure a true timeseries database would be faster, but how often are we making these queries such that the extra speed is worth the complexity?

It sounds like knowledge about how to effectively query Splunk isn't well-distributed; I'll talk with Ryan about scheduling a lunch-and-learn sometime! This document with some ideas about what people are looking for will definitely be helpful.

paulswartz · 2024-02-15T19:29:16Z

+
+- You maintain non-Elixir applications and do not want to write a method of getting metrics into Splunk.
+
+There aren't any libraries in other languages that transmit metrics to Splunk via the HEC.


question: if we're maintaining a library to do this for Elixir, why wouldn't we maintain one for other languages we use?

We predominately use Elixir. So, it would make more sense to offer it as an option. The same ROI would not exist for languages we use less frequently.

paulswartz · 2024-02-15T19:35:53Z

+
+You can argue that we do not need metrics and that adding a means to collect them increases the complexity of our technology stack.
+However, we hope the limitations of a logs-only approach to observability have been made apparent by this RFC.
+Furthermore, we hope that the promised benefits of metrics cause excitement.


suggestion: some other drawbacks offhand:

additional resource usage for the Universal Forwarder

need to keep the UF up-to-date

possibly trickier to manage the connection to Splunk (vs. relying on ECS)

some questions:

how do we monitor the UF?

how does the UF handle being spun up / spun down elastically?

I have addressed the resource usage above. It's a small footprint. Again, this is how Datadog et al. work. The UF has been around forever and is stable. Keeping it up-to-date would just be releasing a new image, which almost always only entails updating the version in the Dockerfile.

The UF isn't different from any other metrics agent; if you're worried that you aren't getting metrics, you write an alert that expects metrics to be sent. You don't need to scale the UF. You only need one.

I thought the UF was implemented as a sidecar to Dotcom? So it's being spun up / spun down as Dotcom is.

paulswartz · 2024-02-15T19:36:26Z

+
+- You maintain applications that aren't written in Elixir or want all of your applications to emit metrics in a standard format.
+
+Every language has a library that emits metrics in the StatsD format; they are straightforward to set up.


question: what other languages do we use which are emitting metrics in StatsD format?

We have a dotcom monitor that is written in Node. It was written in Node because it uses Playwright to run through critical user journeys and emits the time taken as metrics. Using Playwright allows us to reuse the same scenarios for end-to-end tests, load tests, and this monitoring. We also do health checks on our dependencies and emit those as metrics: 0 for down and 1 for up. That allows us to quickly calculate up-time over long periods.

That's the only way it outputs information?

paulswartz · 2024-02-15T19:48:56Z

+
+For example, you want to know how many requests your application served per minute over the last hour. Using logs, you would send a log to Splunk for every request. Dotcom regularly receives 150 requests per second or 9,000 requests per minute. So, if you wanted to look at the span of an hour, you would query 540,000 logs and bin each log by its timestamp. Lastly, you would count each log in each bin. You can do this in Splunk, but it is slow. It gets slower and slower as you increase the time horizon or amount of data because logs are stored as documents optimized to be searched rather than calculated.
+
+If you used metrics instead, your application would keep an internal count of the requests received. Every minute (tuneable per application), it would emit that number and a timestamp to Splunk before resetting itself. So, in a minute, it would emit the number 150 once. That is one request instead of the 9,000 above. Reviewing an hour of data would entail nothing more than looking at 60 numbers. Metrics are a much better approach to answering the question.


question: I tried this just now on Splunk: index=dotcom-prod-application state=set | timechart spam=1m count to get a chart of requests over an hour: "This search has completed and returned 61 results by scanning 526,174 events in 2.013 seconds".

How much faster do we need data to be?

The numbers are to indicate the difference in magnitude. Here, it is 9,000:1. And, again, that 1 is stored as a time series, which is optimized to be read as a number in time. Try the same and extend the window to 7 days as you would for any kind of anomaly detection or manual analysis over time. It takes 2 minutes and 30 seconds to go over 57 million events. For the same span of time, you would look at 10,080 metrics. Again, optimized to be read that way.

You can still optimize over multiple days (or longer); see above comment on when to deploy Dotcom.

paulswartz

suggestion: while it's helpful to track the discussions by putting replies into comments, they should also be incorporated into the body of the RFC. That way, future developers don't need to read through the entire PR history to get the relevant information: they can rely on the text of the document.

paulswartz · 2024-02-26T22:12:00Z

+
+For example, you want to know how many requests your application served per minute over the last hour. Using logs, you would send a log to Splunk for every request. Dotcom regularly receives 150 requests per second or 9,000 requests per minute. So, if you wanted to look at the span of an hour, you would query 540,000 logs and bin each log by its timestamp. Lastly, you would count each log in each bin. You can do this in Splunk, but it is slow. It gets slower and slower as you increase the time horizon or amount of data because logs are stored as documents optimized to be searched rather than calculated.
+
+If you used metrics instead, your application would keep an internal count of the requests received. Every minute (tuneable per application), it would emit that number and a timestamp to Splunk before resetting itself. So, in a minute, it would emit the number 150 once. That is one request instead of the 9,000 above. Reviewing an hour of data would entail nothing more than looking at 60 numbers. Metrics are a much better approach to answering the question.


You can still optimize over multiple days (or longer); see above comment on when to deploy Dotcom.

paulswartz · 2024-02-26T22:12:13Z

+  3. **Traces** - are the representations of individual requests or transactions that flow through a system, which can help identify bottlenecks, dependencies, and root causes of issues.
+
+TID engineers currently make extensive use of logging.
+But, we don't currently have an established mechanism for collecting metrics or traces.


Is there another definition of metrics that talks about their storage and access? The definition you used above doesn't appear to include that.

paulswartz · 2024-02-26T22:17:13Z

+# Motivation
+
+Logs are handy for debugging and other unstructured research like a security audit.
+However, many TID developers have noted that some queries based on logs are incredibly slow, if not impossible, to run.


It's 3:30 - 4am, and it took under 10 seconds for 7 days worth of data; 30 days took 25 seconds. I'm sure a true timeseries database would be faster, but how often are we making these queries such that the extra speed is worth the complexity?

It sounds like knowledge about how to effectively query Splunk isn't well-distributed; I'll talk with Ryan about scheduling a lunch-and-learn sometime! This document with some ideas about what people are looking for will definitely be helpful.

paulswartz · 2024-02-26T22:17:17Z

+But, as noted above, logs are just one of three tools used to observe systems.
+Metrics are optimized for storage and retrieval as numbers in a time series, which makes them cheap to store and fast to retrieve.
+
+By adopting metrics, we can speed up slow queries, decrease the time necessary to generate dashboards and reports, and make previously impossible alerts possible.


We have a alert which does this for another application already: search for "Concentrate: Unusual Metric Drop" in the Splunk alerts.

paulswartz · 2024-02-26T22:17:22Z

+But, as noted above, logs are just one of three tools used to observe systems.
+Metrics are optimized for storage and retrieval as numbers in a time series, which makes them cheap to store and fast to retrieve.
+
+By adopting metrics, we can speed up slow queries, decrease the time necessary to generate dashboards and reports, and make previously impossible alerts possible.


I haven't seen an issue with dashboard queries taking a long time, so I was hoping you had examples of where metrics would measurably improve things. For example, the API and Dotcom dashboards seems to load quickly enough?

paulswartz · 2024-02-26T22:17:26Z

+
+By adopting metrics, we can speed up slow queries, decrease the time necessary to generate dashboards and reports, and make previously impossible alerts possible.
+
+Furthermore, we can decrease costs by reducing the data we send to Splunk.


Are we going to stop logging a message for each request?

paulswartz · 2024-02-26T22:18:08Z

+
+- You maintain applications that aren't written in Elixir or want all of your applications to emit metrics in a standard format.
+
+Every language has a library that emits metrics in the StatsD format; they are straightforward to set up.


That's the only way it outputs information?

paulswartz · 2024-02-26T22:18:12Z

+- You are concerned about your ECS resource usage.
+
+The UF sidecar uses CPU and memory resources.
+Though the footprint is small, you might be worried that increasing your compute usage might affect the rest of your cluster.


It's a sidecar though, correct? So it's additional resources (not many it sounds like), but some.

paulswartz · 2024-02-26T22:18:22Z

+
+You can argue that we do not need metrics and that adding a means to collect them increases the complexity of our technology stack.
+However, we hope the limitations of a logs-only approach to observability have been made apparent by this RFC.
+Furthermore, we hope that the promised benefits of metrics cause excitement.


I thought the UF was implemented as a sidecar to Dotcom? So it's being spun up / spun down as Dotcom is.

ryan-mahoney and others added 11 commits January 29, 2024 10:32

0020-telemetry.md

8aba8bf

add more explanation of telemetry library

74a5d5a

small additions

2a204d4

add splunk information

d4f2461

...

318e939

more info on telemetry

a26ce6f

grammar and splunk uf

9717bb6

other splunk ingestion details

2c36f66

label code block as terraform

c589c76

more negos for cloudwatch

dc83396

clarify that reports emit

daa1ccb

anthonyshull marked this pull request as ready for review February 15, 2024 15:31

paulswartz reviewed Feb 15, 2024

View reviewed changes

paulswartz mentioned this pull request Feb 16, 2024

0020-telemetry.md #20

Closed

paulswartz reviewed Feb 26, 2024

View reviewed changes


		By adopting metrics, we can speed up slow queries, decrease the time necessary to generate dashboards and reports, and make previously impossible alerts possible.

		Furthermore, we can decrease costs by reducing the data we send to Splunk.


		- You maintain non-Elixir applications and do not want to write a method of getting metrics into Splunk.

		There aren't any libraries in other languages that transmit metrics to Splunk via the HEC.


		- You maintain applications that aren't written in Elixir or want all of your applications to emit metrics in a standard format.

		Every language has a library that emits metrics in the StatsD format; they are straightforward to set up.


		For example, you want to know how many requests your application served per minute over the last hour. Using logs, you would send a log to Splunk for every request. Dotcom regularly receives 150 requests per second or 9,000 requests per minute. So, if you wanted to look at the span of an hour, you would query 540,000 logs and bin each log by its timestamp. Lastly, you would count each log in each bin. You can do this in Splunk, but it is slow. It gets slower and slower as you increase the time horizon or amount of data because logs are stored as documents optimized to be searched rather than calculated.

		If you used metrics instead, your application would keep an internal count of the requests received. Every minute (tuneable per application), it would emit that number and a timestamp to Splunk before resetting itself. So, in a minute, it would emit the number 150 once. That is one request instead of the 9,000 above. Reviewing an hour of data would entail nothing more than looking at 60 numbers. Metrics are a much better approach to answering the question.

Uh oh!

Conversation

anthonyshull commented Feb 7, 2024

Uh oh!

paulswartz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anthonyshull Feb 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paulswartz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anthonyshull Feb 26, 2024 •

edited

Loading